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ABSTRACT 

This  report  considers  spectral  estimation  and  extrapolation  techniques 
for  discrete  time,  band  limited  signals,  (l.e.,  signals  whose  bandwidth  Is 
less  than  cycles;  If  T sec.  Is  the  sampling  Interval)  which  are  observable 
only  for  a finite  duration.  The  objective  Is  to  determine  the  spectrum 
(or  power  spectrum)  of  these  signals.  It  Is  shown  that  the  estimated 
spectrum  can  be  Improved  considerably  (over  a perlodogram  or  Maximum  entropy 
spectrum)  by  first  extrapolating  the  given  observations  beyond  the  observation 
Interval.  Also,  we  consider  the  problem  of  extrapolation  of  signal  In  the 
presence  of  noise  or  other  Interfering  signals. 

Several  new  results  and  algorithms  are  presented.  First,  It  Is  shown 
that  some  of  the  existing  extrapolation  methods  for  continuous  signals  when 
extended  to  sampled  data  do  not  converge  to  the  exact  original  time-unlimited 
signal.  Rather,  one  only  expects  to  get  a minimum  norm  least  squares  estimate. 
And,  we  find  that  Papoulls' [8]  Iterative  extrapolation  algorithm  Is  a special 
case  of  a gradient  algorithm  with  linear  convergence.  It  Is  shown  that  an 
Infinite  extrapolation  matrix  Introduced  In  [10]  does  not  exist  and  Is  Ill- 
conditioned  at  best  when  approximated  to  a finite  matrix.  The  new  extrapolation 
algorithms  Include  a discrete  prolate  spheroidal  wave  function  (PSWF)  expansion, 
a conjugate  gradient  Iterative  algorithm,  a mean  square  extrapolation  filter 
and  a recursive  Kalman  filter  type  extrapolator.  The  latter  two  algorithms 
also  consider  the  noise  statistics  In  extrapolation  of  the  signal.  Several 
examples  are  given  and  comparisons  are  made. 
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I.  INTRODUCTION 

Spectral  estimation  refers  to  the  problem  of  estimating  the  spectral 
density  function  of  a stationary  random  signal  which  Is  observable  only 
over  a finite  duration.  For  a deterministic  signal  it  implies  estimation 
of  its  magnitude  spectrum.  In  either  case,  if  the  signal  were  known  over 
the  infinite  interval,  the  Fourier  transform  of  the  signal  o»  its  auto- 
correlation would  immediately  yield  the  spectrum.  Thus,  any  estimated  spectrum 
is  equivalent  to  specifying  the  signal  or  its  autocorrelation  outside  the 
observation  interval--1 .e. , its  extrapolation. 

In  this  report  we  consider  several  algorithms  for  extrapolation  and 
spectral  estimation  of  discrete  time  signals.  First,  we  briefly  review 
the  maximum  entropy  (ME)  or  the  linear  predictive  autoregressive  (AR) 
method,  and  some  iterative  and  matrix  inverse  based  extrapolation 
algorithms  developed  recently  by  Papoulls  [8],  Sabri  and  Steenaart  [10], 
and  Cadzow  [11]. 

The  new  results  presented  here  are  as  follows. 

1)  Papoulls'  iterative  algorithm  applied  in  discrete  time  domain  converges 
to  an  extrapolated  signal  which  is  a minimum  norm  least  squares 
type  solution.  It  is  seen  to  be  a special  case  of  a one  step  gradient 
algorithm,  and  has  linear  convergence.  The  convergence  of  this  can 
be  Improved  by  suitably  modifying  It  to  a steepest  descent  algorithm. 

2)  Sabri  and  Steenaart  [10]  have  reformulated  Papoulls1  Iterative  algorithm 

In  terms  of  an  extrapolation  matrix  operator  which  yields  the 
extrapolated  signal  when  It  operates  on  the  given  time  truncated 
signal.  It  Is  proven  that  the  Infinite  operator,  E,,,  does  not  exist, 
but  Its  finite  truncation  EN  exists,  but  It  is  ill-conditioned. 
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3)  It  Is  known  that  a continuous  (time)  band  limited  signal  given  over 

a finite  observation  interval  can  be  extrapolated  exactly  outside  this 
Interval  by  means  of  Prolate  Spheroidal  Wave  Functions  (PSWF).  We 
show  that  for  the  discrete  time  case  a similar  expansion  arises  when 
we  consider  the  minimum  norm  least  squares  extrapolation. 

Then  we  present  three  other  algorithms  which  are  as  follows: 

4)  Conjugate  Gradient  Iterative  Extrapolation 

5)  Minimum  Mean  Square  Extrapolation  Filter 

6)  Recursive  Least  Squares  Extrapolation  Filter 

The  conjugate  gradient  method  is  an  iterative  algorithm  which  yields 
a psuedo  Inverse  extrapolation  operator.  Compared  to  the  earlier  Iterative 
methods  [8-11],  this  algorithm  converges  quite  rapidly.  The  minimum  mean 
square  extrapolation  algorithm  is  designed  for  applications  where  the 
observed  band  limited  signal  Is  contaminated  by  wideband  white  noise.  It 
yields  a simple,  Wiener  filter  type,  extrapolation  operator  which  requires 
Inversion  of  a matrix  whose  size  is  equal  to  the  number  of  samples  In 
the  observed  signal.  No  Iterations  are  required  here  and  the  algorithm 
Is  shown  to  reduce  to  the  matrix  Inverse  algorithm  of  Cadzow  [11] 
as  the  additive  noise  power  goes  to  zero.  Finally,  the  recursive 
least  squares  algorithm  Is  a Kalman  filter  based  method  where  the 
extrapolated  signal  estimate  Is  updated  recursively  as  a new  observation 
sample  arrives.  The  latter  two  methods  are  applicable  In  the  presence 
of  noise  and  yield  stable  results.  Finally  these  algorithms  are  shown  to 
be  applicable  to  problems  where  one  needs  to  discriminate  as  well  as 
extrapolate  an  Interfering  signal  and  a desired  signal. 

Several  examples  are  considered  to  compare  the  various  algorithms. 


Mi 
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This  could  be  written  as 

S(f)  = 


> a £ 
m=0  m 


-j2Trfm 


(6) 


where  the  am  and  am  are  related  by 


a * a 
-m  m 


m1n(p-m,m) 

i Wk 

k*max[0,-m] 


(7) 


The  coefficients  (am>  are  detennlned  by  solving  (4)  and  (6)  which  Is  equivalent 
to  solving  (9)  below.  Alternatively,  the  {u^}  could  be  characterized  as  an  AR  process 


Ui  * ) a Ul 

k nuk-n 


+ eL 
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where  B2  * E[ek2].  By  writing  the  Yule-Walker  equations  for  (8)  It  Is  a 
simple  matter  to  show  that  {am}  are  obtained  by  solving  a set  of  simultaneous 
linear  equations* 

Ra  = -621_  * 82  ~ V(R  )^i 


where  a and  1_  are  (p+1)  x 1 vectors  and  R is  a (p+1)  x (p+1)  covariance 
matrix  with  entries  corresponding  to  covariances  on  the  window  W,  1.e.» 


Is  positive  and  (8)  Is  asymptotically,  a stationary  random  process. 
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Strictly  speaking,  the  covariance  values  {rm,  m e W)  should  be  known 
exactly.  In  practice,  one  only  knows  data  values  on  a finite  window. 

Then,  the  covariances  could  be  estimated  as* 

, M-|m| 

= “kV|m|’  "eW  (10) 

where  M is  the  size  of  the  data  window.  For  large  M » p,  reasonable 
estimates  of  {rj could  be  expected. 

Note  that  this  method  does  not  require  {u^}  to  be  bandlimlted  (with 
respect  to  the  Nyquist  rate).  Also,  the  spectral  density  function  is,  in 
view  of  (5)  and  (6),  an  all  pole  model.  Thus,  if  the  given  observations 
were  of  the  form 

*k  ■ uk  * nk  (11) 

where  nk  is  a white  noise  process  or  another  signal  (e.g.  clutter  noise 
which  could  be  modeled  by  an  AR  process),  the  spectrum  of  {yk>  would  not 

be  an  all  pole  model  and  may  have  to  be  approximated  by  a very  high 
order  all  pole  model. 

Example  1(a):  Although  there  are  many  examples  where  the  maximum  entropy 
method  could  be  applied  successfully  [3*5,7]  we  consider  a case  where 
It  does  not.  We  assume  the  observations  to  be  given  by 

*k  ■ sk  * s + nk  (12) 

where  sk  represents  a bandllmited  signal  whose  spectrum  lies  In  the 
Interval  [f2,f3]  and  [-f2,-f3l  and  ck  Is  an  Interference  signal  band- 
limited  In  the  Interval  [-fj.fj]  and  nk  Is  a white  noise  process. 

*ln  estimating  rm,  the  divisor  of  M,  rather  than  M-|m|  Is  recommended . 
Although  this  results  In  a biased  estimate  of  rm,  it  ytelds  a positive 
definite  sequence  {rm>  so  that  R Is  positive  definite  and  the  resulting 
spectra  Is  positive.  See  Parzen  [15]  for  details. 
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Figure  1(a)  shows  the  spectra  of  the  various  signals.  Figure  1(b)  and  1(c) 
show  the  Interference  signal  ck  and  the  actual  signal  s^,  modeled  as 

sk  = 1.69  S1n( . 39irk). 

Figure  1(d)  shows  the  17  samples  of  the  available  observations. 

The  signal  to  Interference  signal  (to  be  called  clutter)  ratio,  wMch 
Is  defined  as 


Peak  to  Peak  Value  of  Signal 

SCR*  = 20  log1Q r.m.s.  value  of  Gutter-'  * (13) 

Is  -4.1  dB  and  the  signal  to  noise  ratio,  SNR,  defined  similarly  Is  19  BB. 
Figure  1(e)  shows  the  maximum  entropy  spectrum  estimate.  A peak  Is 
expected  at  the  position  marked  by  the  arrow.  At  this  point  the  signal 
estimate  is  30  dB  below  the  clutter  peak  and  is  indistinguishable  from 
the  Interfering  signal.  Figure  1(f)  shows  the  spectrum  estimated  by 
directly  evaluating  the  Fourier  spectrum  (i.e.  the  periodogram)  as 


S(f)  = 17 


Equation  (14)  can  be  evaluated  approximately  by  discretizing  the  variable 
f and  using  a fast  Fourier  transform  algorithm.  The  spectrum  of  Figure  1(f) 
Is  the  result  of  a 256  point  FFT.  We  note  that  both  of  these  estimates 
are  unsatisfactory*  We  will  see  that  the  new  algorithms  Introduced  here 
Improve  the  estimated  signal  spectrum  considerably. 


♦Note  that  for  random  signals,  the  periodogram  Is  an  Inconsistent  estimate. 
Windowing  techniques  may  be  used  to  Improve  the  spectrum  estimate  In  the 
sense  that  It  would  be  a consistent  estimate  of  a smoothed  version  of  the 
original  spectrum.  In  this  example,  windowing  did  not  Improve  the  situation 
In  so  far  as  the  signal  was  concerned. 


I ykexp(-j2irfk) 
k=-8  K 


1 < f * 1 


04) 
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III.  EXTRAPOLATION  OF  BANDLIMITED  SIGNALS 
3.1  Continuous  Time  Signals 


Suppose  we  have  a continuous,  bandllmlted  function  f(t)  so  that  its 
Fourier  transform  satisfies 

F(<d)  = 0 for  |ui|  > a . (15) 

Let  gQ(t)  be  a time  limited  segment  of  f(t)  which  is  available  as  a 
noise- free  observation,  viz.. 


g0U)  • 


ff(t)  , |t|  < T 


, 1 1 1 > T 


The  problem  Is  to  extrapolate  g0( t)  outside  the  interval  [-T,T].  This 
is  the  classical  problem  of  extrapolation  of  analytic  functions.  The 
existence  of  a unique  solution  can  be  established  by  observing  that 
bandllmi tedness  of  f(t)  Implies  It  is  analytic.  This  means  all  its 

derivatives  exist  and  are  bounded  so  that  from  the  Taylor  series  expansion 

.2 

f(T+A)  = f(T)  + Af (T)  + f"(T)  + ...  (17) 

one  can  evaluate  f(t)  outside  [-T.T].  In  practice,  (17)  Is  not  very 
useful,  because,  not  only  does  the  series  have  to  be  truncated,  but  also, 
the  evaluation  of  various  derivatives  is  a noise  sensitive  process.  An 
alternative  algorithm  suggested  by  Slepian,  et  al.  [14],  uses  a series 


expansion 


fit)  = l a <J>  (t,To) 
n*0  n n 
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where  {<j>n(t,To)}  Is  a special  set  of  complete  orthonormal  bandllmlted 
functions,  called  the  prolate-spheroidal  wave  functions  (PSWF),  which  are 
defined  for  all  t.  The  coefficients  (an>  can  be  evaluated  as  projections 
of  the  known  function  9g(t)  on  the  basis  functions  {$n>.  Once  {an>  are 
known,  the  right  side  of  (18),  considered  valid  for  all  t, gives  the 
extrapolated  signal.  In  practice,  this  method  also  suffers  from  noise 
limitations  and  errors  due  to  truncation  of  the  series.  Moreover  It 
is  extremely  difficult  to  accurately  generate  the  basis  PSW  functions 
so  that  extrapolation  In  a practical  situation  is  quite  hopeless.  For  a 
simDle  example,  see  Frieden  [19]. 

Recently,  Papoulls  [8]  has  introduced  an  iterative  scheme  that  appears 
to  do  better  than  the  PSWF  expansion  method.  The  algorithm  has  the 
following  steps.  The  first  step  is  to  compute  the  Fourier  transform  of 
gQ(t)  as  Gq(co)  and  define 


0(«).  M < o 


o , M > o 


Compute  f^(t),  the  Fourier  inverse  of  F^oj)  and  let 


f(t)  , |t|  <T 
^(t),  |t|  > T 


(19) 


(20) 


Then  compute  G^U)  = F[g^(t)]. 

This  Is  the  first  step  of  the  Iteration.  At  the  nth  step  form  the 
function 


F„<“! 


Gn_i  (<*>>.  1^1  <0 
0 , |w|  > a . 


- < 


(21) 
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Find  F~  [Fn(u>)]  and  form 


g„{*) 


f(t)  , |t|  < T 

yo , jti  > t 


Papoulis  has  theoretically  shown  that  fn(t)  converges  to  f(t)  as 
n -*•  « . If  we  define  a band-limiting  operator  as 


Bf (t)  * f(t)  © ((sinot)/iTt) 


where  (sinat)/irt)  represents  the  impulse  response  of  a low  pass  filter,  and 
we  define  a time-1 imiting  operator  as 


Df(t) 


f(t)  . |t|  <T 
0 . 1 1 1 > T 


D = I - D 


where  I = identity  operator 
then  the  foregoing  algorithm  can  be  written  as  [10] 

gn(t)  = 9o(t)  + HVi(t) 

fn(t)  - f,(t)  + Gfn-1(t) 


H * DB,  G * BD 


f (t)  = l Gk  f.(t)  . G°  ■ I 
n k«0  ' 


9n(t)  - I Hk  90(t) 
n k-0 


-V"  .... — — . . " . 
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Elther  9n(t)  or  -fn (t ) may  be  considered  as  the  extrapolated  signal. 

3.2  Extension  to  Discrete  Time  Signals 

Sabrl  and  Steenaart  [10]  have  suggested  a discrete  version  of  this 
algorithm,  as  follows. 

Let  y(k)  be  a discrete,  bandllmlted  signal  so  that  Its  Fourier  trans- 
form (l.e. , Z transform  evaluated  on  the  unit  circle)  defined  as 

Y(f)  * l y(k)exp(-j2irfk)  , -y<f<i  (28) 

k=-00 

satisfies  the  relation* 

Y(f ) - 0 , j > |f|  > a (29) 

We  are  given  a set  of  time  limited,  noise  free  observations 

y(k)  , -M  < k < M 

g (k)  = • 

0 0 , otherwise  . 

Given  {gQ(k)},  the  problem  is  to  find  an  estimate  of  y(k)  outside  the 
Interval  [-M.M].  Following  section  3.1,  we  define  infinite  vectors 

y * [...y(-k)...y(-l),y(o),y(l),...,y(k)...]T 

T (30> 

gn  * [• . .9n(”k)» • •9n(-l ) *9n(0) »9n(1 ) »• • • »9n(k) • • •] 

where  gQ  & [0,0., ,,0  g0(-M),g0(-M+l) 9Q(-1 ) .90(0) ,90(1 ) • — 9(j(M) .0,0. — 03  • 

We  also  define  a band-limiting  operator  L,  and  a time-limiting  operator  W, 
as  Infinite  matrices 

♦this  Implies  y(k)  has  been  oversampled  with  respect  to  its  Myqulst  rate. 

This  occurs  quite  often  when  a system  observes  signals  over  a wide  bandwidth. 


in  the  limit  as  n -*■  » , we  get 
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In  practice,  the  Infinite  matrix  operator  G Is  replaced  by  a finite 
matrix,  of  size,  say,  NxN  where  N » (2M+1).  Later  In  this  section  we  will 
show  that  (I-G)  Is  singular  for  N = <*>,  but  Its  finite  approximation  (N<») 

Is  non  singular. 

Existence  and  Convergence 

Following  Papoulls,  It  can  be  shown  that  the  above  iterative  algorithm 
satisfies  the  Inequality 
■% 

|Y(f)  - Gn_.,(f)|2df  > | Y (f ) - Gn(f)|2df  (38) 

which  says  that  the  mean  square  error  is  reduced  at  each  step.  However,  the 
extrapolated  signal  need  not  converge  to  the  original  signal  y(k)  because 
the  time  limited  discrete  signal  does  not  have  the  analytlclty  property  that 
the  continuous  signal  has.  Indeed,  as  we  show  in  the  next  section,  the  above 
(discrete  extrapolation)  algorithm  converges  to  a least  squares,  minimum 
norm  solution  associated  with  the  solution  of  the  equation 

WLy  = gQ 

In  terms  of  computational  complexity,  the  Iterative  algorithm  requires 
about  4nNlog2N  real  operations  (one  operation  = one  multiplication  and  one 
addition),  where  N is  the  size  of  the  extrapolated  vector  (and  Is  much  larger 
than  M)  and  n is  the  number  of  Iterations. 

If  the  extrapolation  matrix  E„1s  used,  then  once  It  has  been  computed. 

It  requires  ^(2M+1 )(N-2M-1 ) operations  to  evaluate  the  extrapolated  signal. 
However  a large  (NxN)  matrix  which  Is  111  conditioned, has  to  be  inverted. 

(see  next  section) 
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IV.  EXTRAPOLATION  OF  DISCRETE  TIME,  BANDLIMITED  SIGNALS 

Before  proceeding  to  prove  several  results  related  to  extrapolation 
of  discrete  signals  we  first  consider  several  definitions.  Let  A be  an 
arbitrary  mxn  matrix  and  consider  the  system  of  equations 

Ay  * z (39) 

where  y and  z are  nxl  and  mxl  vectors  respectively. 

4.1  Definitions 

Definition  1:  Least  Squares  Solution 

A least  squares  solution  of  (39),  denoted  by  y,  is  such  that 


| | z - Ap| |2  = (z-Ay)T(z-Ay) 


(40) 


Is  minimum.  This  solution  must  therefore  satisfy  the  equation 


ATAy  = ATz 


(41) 


If  ATA  Is  nonsingular  (l.e.  n < m and  rank  of  A Is  n)  then 


y * (ATA)  ATz 


(42) 


Is  the  least  squares  solution  and  Is  unique.  If  m * n and  A Is  nonsingular, 
then 


y * y ■ A ^z 


(43) 
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If  m < n,  then  ATA  Is  necessarily  singular  and  has  rank  at  mast  m.  Then 
(41)  does  not  have  a unique  solution. 

Definition  2:  Minimum  Norm  Least  Squares  Solution 

Let  y+  denote  this  solution.  Then  y+  must  be  that  solution  of  (41) 

O 

which  has  the  minimum  norm  ||y||  . Thus 

y+  = min{||y||2;  ATAy  = ATz)  (44) 

y 

Clearly,  if  rank  ATA  Is  n then  y+  = y.  The  minimum  norm  Is  simply 
a constraint  that  makes  the  least  squares  solution  unique  for  an  arbitrary  A. 
Definition  3:  Pseudo  Inverse 

We  call  A+  the  pseudo  Inverse  of  A[18],  if  for  every  equation  (39), 
the  associated  minimum  norm  least  squares  solution  is  given  by 

y+  = A+z  (45) 

This  pseudo  Inverse,  also  called  the  generalized  inverse,  satisfies 
the  relations 

AA+  = (AA+)T 
A+A  = (A+A)T 

. (46) 

AA  A = A 

A+AA+  - A+ 

When  rank  A ■ n , 


r 
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A+  = (ATA)  AT 


If  rank  A = m,  then 


= at(aat)  1 


(47) 


(48) 


Definition  4:  Singular  Value  Expansion  [21] 

In  general  an  explicit  expression  for  A+,  of  the  type  of  (47)  or  (48) 
is  not  available.  However,  A+  can  be  expressed  as  an  expansion.  Consider 
the  eigenvalue  problems 


■ Vk 

M\  • Vk 


(49) 


where  k = l,2,...,p  and  p is  the  rank  of  A.  The  vectors  and  are  of 
sizes  nxl  and  mxl  respectively.  Since  ATA  and  AAT  are  non-negative  matrices, 
these  eigen-vectors  exist  and  can  be  orthonormal i zed  so  that 


ly^  = 6k,£ 


V*  = 6k,Jl 


(50) 


From  this, one  can  express  the  rectangular  matrix  A by  the  expansion,  called 
the  singular  value  expansion,  as 
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A " j,xkVk 


where  > 0,  are  called  the  singular  values  of  A. 
The  pseudo  inverse  A+  can  now  be  written  as 


(51) 


A*  ■ A 


(52) 


4.2  Properties  of  L 

Now  we  consider  some  useful  properties  of  the  low  pass  filter  operator 
defined  in  (31). 

Property  I:  L is  a symmetric  operator,  i.e.,  L * LT.  This  follows 

from  its  definition. 

Property  II:  The  Fourier  spectrum  of  L is  given  by 


*(f) 


It  0 < | f | < a 
0,  otherwise. 


(53) 


where  - £ 5 f < 7.  This  is  obvious  since  L is  the  Toeplltz  matrix 
formed  by  the  Fourier  Inverse  of  t(f),  the  lowpass  filter  transfer  function. 

Property  III:  Let  S be  a (2M+l)x°°  matrix  operator  whose  elements 
are 


Mj 


1,  1«j«0*l,*2 ±M 

0,  otherwise 


(54) 


Basically  S selects  (2M+1)  elements  from  an  Infinite  vector.  Consider 
the  (2M+1 )x(2M+l ) matrix 


L - SIS' 


(55) 


I 


t 
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Then 


*1,J  = *1-J 


Sln2ira(1-j)  . 


- M < 1 , j < M 


(56) 


Also 


STS  = W (57) 

where  W Is  defined  in  (31). 

Property  IV:  The  operator  L is  idempotent,  i.e., 

L2  = L (58) 

This  is  obvious  because  ideal  low  pass  filtering  a signal  once  Is  the 
same  as  doing  It  twice,  I.e., 

Ly  * L(Ly) 

Note  this  implies  the  spectrum  (or  eigenvalues)  of  L must  be  composed 
of  zeros  and  ones  only  [see  (53)]. 

A 

Property  V:  For  every  M < °°,  L Is  positive  definite.  This  follows 


ta 

x Lx* 


1 1 


x* 
m-n  n 


l l (f°  eJ2ir<m-n>fdf)xmX 

m n J-o 


* 

n 


by  noting 


r 2df 


5 rix^oiv.  vfisjviw 


>0,  If  M < 


If  M = »,  then  x = e"'’27^  gives  XM(f)  = 6(f-£)  so  that 


T*  T 

x Lx*  = x Lx* 


1,  \Z\  < a 

0,  |5|  > a 


and  is  not  positive  definite.  Thus,  all  the  eigenvalues  of  L are  positive 


X (L ) >0,  M < °° 


Let  X (L)  denote  the  largest  eigenvalue  of  L.  Then 
max 


X (L)  < 1 , M < °° 
max'  ' 

=1,  M = » 


Thus,  for  any  finite  M,  the  eigenvalues  of  L are  bounded  In  the  Interval 
(0.1)  i.e.. 


0 < X(L)  < 1 , M < “ 


To  prove  (62)  we  note 


Xli)  6 max 
max  £}  x”x* 


From  (59)  we  can  write 


max  / w \ r 1 


cm** 

max  -175 

{xm}f  i/2|xM(f)|2<lf 


Since,  for  M < »,  XM(f)  Is  the  Fourier  spectrum  of  a time  limited  signal. 

It  cannot  be  zero  on  any  finite  Interval.  Hence 

f°  I XM(f ) 1 2df  < fV2  X (f)  2df,  V M < -,  a < \ 

J-o1  M 1 ■* -1/2  m 

- * M o 

or  T I x l x < l |x | , V M < «. 
m n m m"n  n m-M  m 

When  M » °°,  one  could  maximize  (64)  by  choosing  a bandllmlted  signal  so  that 
the  above  Inequality  will  become  an  equality.  This  proves  (62). 

4.3  Iterative  Extrapolation 

With  the  above  definitions  and  properties  we  are  now  ready  to  prove 
the  following  results.  Let  y(k),  k-0,*l,...  be  a discrete  time  bandllmlted 
signal  as  defined  In  (28)  and  (29). 

Furthermore,  let  this  signal  be  observed  without  any  noise  over  a 


finite  Interval  and  define 


z(k)  = y(k) , - N < k < M 


If  z denotes  a (2H*l)xl  vector  and  y Is  the  Infinite  vector  of  (y(k)},  then 


z * Sy 
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SI  nee  y Is  band! 1ml ted.  It  must  satisfy 


Ly  ■ y 


so  that  we  can  write 


z • SLy 


Theorem  1 : Minimum  Norm  Least  Squares  Extrapolation  Theorem 
The  Iterative  solution  [see  (32)  to  (37)  and  (57)] 


fq+l  f-j  + q = 1,2,... 

G = L(I-STS)  = L(I-W) 
f1  = Lg0  = LSTz 


converges  to  the  minimum  norm  least  squares  solution  y+  of  (66).  Moreover, 
(67)  Is  a special  case  of  a gradient  algorithm  associated  with  the  minimum 
norm  least  squares  optimization  problem. 

Proof:  An  Iterative  gradient  algorithm  associated  with  the  minimum  norm 
least  squares  solution  of  the  general  equation  (39)  Is 


yq+l  ’ yq  + a A 


I ATz  ♦ (I  - 1 *TA)yq 


It  is  known  that  yq  converges  to  y as  q .-*■  »,  under  the  following  conditions  [12]: 
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a)  0 < 1 < 


* 


(70) 


b)  The  Initial  guess  yQ  must  lie  In  the  range  space  of  ATA  e.g.,yQ  * 0. 
From  (66),  letting 


A = SL 


(71) 


we  get 


ATA  = LTSTSL  = LWL  (72) 

ATz  = LTSTz  = LSTz  = f1  (73) 


Hence  (69)  becomes 


^q+1  o ^1  + 


- i LWL 


(74) 


Now  letting 


y0  ■ o 


and  noting  that  f^  Is  bandllmlted  l.e., 

Lf-|  - (75) 

It  Is  easily  verified  by  Induction  that  (yq)  Is  a bandllmlted  sequence  l.e.. 
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Lyq  = yq.  q=l  ,2,3. . . 


Using  this  In  (74)  we  get 


Vl  ' o fl  + *q  - S'  LMl,q 


■?fl  * (I  " o LW)yq 


(76) 


(77) 


For  o = 1 , (77)  becomes  the  same  as  (67).  Now  It  remains  to  show  that  this 
algorithm  converges  for  a = 1.  From  (70)  and  (72)  this  requires  us  to  find 
the  largest  eigenvalue  of  LWL.  Now 


max' 


max 


(ATA) 

■ W«T> 

■ Wsl/> 

■ wc> 

< 1, 


y h ‘ 


where  we  have  used  Property  VI.  Therefore,  convergence  of  (77)  Is  achieved 
whenever 


o < 1 < 2 < (78) 

* ■ WL> 

Hence  for  a ■ 1,  (77)  converges.  This  completes  the  proof  of  Theorem  1. 

An  Interesting  question  raised  by  the  foregoing  result  Is  "What  Is  the 
optimal  value  of  o?"  In  other  words,  we  want  to  find  the  "steepest  descent" 
for  the  gradient  algorithm.  Defining  the  error  vector  at  Iteration  step  q as 
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Cq  = y " yq 


and  noting  that  can  also  be  written  as 


f1  = Lg0  = LWy  = LWLy 


we  obtain  from  (74) 


eq+l  ^ " a LW*-)eq 


- (I  - i LWL)  e0,  e0-y 


This  shows  the  convergence  rate  of  this  extrapolation  algorithm  Is  linear. 
Slow  convergence  of  this  algorithm  has  also  been  noted  experimentally 
by  us  and  by  Papoulis  [8]  and  Sabrl  et  al  [10].  Since  this  Is  a gradient 
algorithm,  convergence  can  be  Improved  by  adjusting  o at  every  iteration. 
The  optimal  value  Is  given  by 


"qTATAhq 


where  hq  is  the  gradient  at  step  q,  defined  as 


A * SL, 


hq  = A (*-%) 


This  requires  additional  computations  at  every  Iteration  step.  If  a constant 
value  of  a Is  desired.  It  Is  given  by  [24] 


°opt''  = 2 5 2/W  (85> 

From  our  foregoing  analysis  we  conclude  the  following  about  Papoulls' 

Iterative  algorithm. 

1.  The  solution  converges  to  a minimum  norm  least  squares  solution.  Note 
that  continuous  version  of  the  algorithm  converges  to  the  original  band 
limited  signal  y(t),  as  proven  by  Papoulls  [8].  This  reinforces  the  fact 
that  time  limited  discrete  observations  of  a band  limited  signal  need  not 
give  Its  exact  extrapolation. 

2.  The  algorithm  Is  a gradient  algorithm.  Hence  Its  convergence  Is 
linear  and  slow.  It  could  be  Improved  by  going  to  the  steepest  descent 
algorithm. 

4.4  The  Extrapolation  Matrix 

Now  we  consider  the  extrapolation  matrix  suggested  by  Sabrl  and  Steenaart 
[10].  This  Is  the  doubly  Infinite  matrix  defined  as  [see  eqns.  (32)  to  (36)] 


E 
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G = L(I-W) 

In  a practical  situation  the  matrix  G Is  truncated  to  a finite,  but  large, 
NxN  matrix,  say  G^  defined  as 


6N  s ^ 

and  the  corresponding  extrapolation  matrix  Is 

en  = 1 

We  Intend  to  show  that  for  every  finite  N,  EN  exists.  However,  E,,,  does 
not  exist.  Thus  as  N goes  to  infinity  the  sequence  {E^}  becomes  an  ill- 
conditioned  set  of  matrices. 

Lenina  1 : For  every  finite  N,  the  matrix  defined  as 

PN  . I - L„  * LnW„  (88) 

Is  nonsingular.  At  N * <*>,  PN  is  singular. 

Proof:  From  Property  V,  the  finite  NxN  matrix  Is  positive  definite. 

Now  consider  the  symmetric  matrix 

CN  4 P<h 

“ LN  " LN2  * lnhnln ' *89' 

Since  all  the  eigenvalues  of  LN  lie  In  the  Interval  (0,1)  we  have 
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; 


V-V  * Xk2^LN^  < Xk^LN^ 

Therefore,  for  any  Nxl  vector  x, 

T T 2 

X L^x*  > X'L^X* 


(90) 


Also 


xTlnwnlnx*  = 


M 

l 


M 

l 


m=-M  n=-M 


X £ (^)y  * > 

xnrm,nxn 


where  U^}  are  the  (2M+1  )x(2M+l ) elements  of  LN2,  which  is  positive 
m,n  n 

definite.  Clearly,  then  CN  is  positive  definite.  Hence  PN  = exists 

and  Is  nonsingular. 

At  N = <*>,  Iflis  singular.  Consider  the  eigenvectors  of  the  equation 


LWLx  = Ax 


(91a) 


Since  LWl  Is  symmetric  and  is  of  rank  (2M+1),  there  exists  an  x such  that 

LWLx  =0,  x^O  (91b) 


Moreover,  for  every  such  vector  there  exists  a band-limited  x.^ 

Lx  * x 

which  is  also  a solution  of  (91a).  Thus,  for  all  such  x we  have 


H 


I 
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xTPx  = x^PLx  = xTLx  - xTL^x  + xTLWLx 
= x^Lx  - xTLx  + xTLWLx 
= xTLWLx 
= 0, 

Thus  P Is  singular. 

4.5  The  Generalized  Inverse 

Having  noted  that  the  foregoing  approaches  give  a minimum  norm  least 
squares  solution,  one  may  attempt  to  find  it  directly.  We  recall  that  the 
given  system  of  equations  is 

SLy  = z (92) 

Defining  A = SL  to  give 

aat  * sllV 

= SLST  (93) 

A 

* L 

We  note  that  L Is  positive  definite  (Property  V).  Hence  from  Definition  3 and 
Eqn.  (48)  we  can  write  directly  A+  = AT(AATf1  which  gives  the  extrapolation 
matrix 

Ec  * lV(SIST)_1  (94) 


and 
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i 


y+  = LTST(SLST)_1z  (95) 

This  form  of  solution  was  obtained  for  the  extrapolation  problem  by  Cadzow 
[11]  by  a different  route.  This  method,  we  believe,  is  easier  and  more 
direct.  We  note  that  while  the  extrapolation  matrix  E^  did  not  exist,  the 

A 

extrapolation  matrix  Ec  exists.  Moreover,  L is  only  a (2M+1  )x(2M+l ) 

Toeplitz  matrix  so  that  its  dimensionality  is  much  smaller.  However,  as  M 

a 

becomes  large  or  for  certain  combinations  of  M and  o,  L could  be  111 
conditioned.  Experimentally,  the  ill  conditioning  can  be  reduced  by  adding 
a small  diagonal  term  to  L.  This,  however,  will  degrade  the  extrapolated 
estimate. 

4 . 6 Discrete  Prolate  Spheroidal  Wave  Functions  and  Singular  Value  Expansion 
In  an  earlier  section  we  had  mentioned  that  a continuous  band-limited 
signal  could  be  extrapolated  outside  its  observation  Interval,  exactly,  via 
the  PSWF  expansion.  For  the  case  of  discrete  signals, a similar  expansion 
is  possible  for  the  minimum  norm  least  squares  extrapolated  estimate.  This 


is  achieved  vi*  the  singular  value  expansion  described  in  Definition  4. 
Papoulls  and  Bertram  [20]  have  Introduced  the  discrete  PSWF  earlier  for 
realization  of  digital  filters  whose  Impulse  response  Is  an  all-zero  model. 
However,  they  have  not  shown  any  extrapolation  properties  of  these  PSWFs. 
Here  we  introduce  the  discrete  PSWFs  which  also  extrapolate  a discrete 
band  limited  signal  (known  over  a finite  duration)  to  an  Infinite,  minimum 
norm  least  squares  signal. 

Following  Definition  4 for  A - SL,  we  consider  the  eigenvalue  problems 


AT A4>k  = LWUk  = LSTSUk  = Xk<j>k,  - M < k < M 


AAT^k  = Uk  = SLST^k  = X^,  - M < k < M 


where  Xk  > 0,  and  (<t>k)  are  « x 1 and  {^k>  are  (2M+l)xl  orthogonal  vectors  l.e.. 


*k  *£  = 6k,£ 

\\  = 6k,A 


From  (96),  4>k  must  be  a band-limited  signal  satisfying  the  condition 


lh  = *k 


because  <j>k  = ( 1 /Xk )L ( WL4>k ) , and  Lz  is  band  limited, Vz.  Now  define 


£k  " SL*k  - S*k 


(100) 


Then  (96)  gives 


LS'sk  - Vk 


* ¥k 


(101) 


This  shows  £k  satisfies  the  same  equation  as  i|»k<  Hence  £k  and  4»k  must 
be  proportional  l.e. 
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Using  the  orthogonality  condition  (98)  and  the  eqns.  (99),  (TOO)  and  (97) 
we  find 


or 


002) 


giving 


, Xk  > 0,  - M < k < M (103) 


Also  from  (96),  this  yields 


- M < k < M 


004) 


Euqatlon  (103)  states  that  the  (2M+l)xl  vector  Is  simply  obtained  by  selecting 

-1/2 

the  (2M+1 ) elements  4k(m),  - M < m < M}  of  4>k  and  scaling  them  by  Xk  . 

Eqn.  (104)  Is  remarkable  In  that  the  »xl  vector  4>k  is  obtained  by  simply  low 

-1/2 

pass  filtering  the  sequence  (i|/k(m)}  and  scaling  the  result  by  Xk  . This 
means  <|>k  Is  the  extrapolation  of  <|>k,  obtained  by  simple  low  pass  filtering  and 
scaling.  Also  noteworthy  Is  the  fact  that  the  sequence  (<J>k(m),  - » < m < »} 

Is  orthogonal  over  the  Interval  - M < m < M as  well  as  over  the  Infinite  Interval. 
This  property  Is  similar  to  that  of  the  continuous  PSWFs.  The  (2H+l)xl  vectors 
are  easily  obtained  by  solving  the  eigenvalue  problem  of  (97)  l.e. 
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- M < n,  k < M (105) 


Once  {^k}  are  obtained,  {d>k>  are  given  by  (104)  l.e.. 


■ j-  JM  *k(m)-  (l06) 


combining  (105)  and  (106)  we  find 

( Aj^n),  - M < n < M, 


*k(n)  = 


(107) 


where  - M < k < M.  The  extrapolated  signal  is  obtained  by  applying  (52) 
as 


+ * + 
y = A z 


or 


M a 


y+(m)  = l -7— 4>k(m),  Vm 

k=-M  A.  K 


(108) 


where 


T M M 

“k'l’k'-  lA1"11.'  I ♦kWrM 

m=-n  m=-n 


(109) 
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The  functions  {^R(n)}  and  (<|>k(n)}  have  been  obtained  by  Papoulls  [20]  and  are 
called  the  discrete  PSWF.  However,  their  usefulness  In  obtaining  minimum  norm 
least  squares  extrapolation  has  not  been  noted  earlier.  Our  extrapolation 
algorithm  requires  the  following  steps.  First  calculate  (2H*1)  orthonormal  1 zed 
vectors  (each  of  size  (2M+1))  by  solving  (105).  Next  obtain  the  (2M*1) 

Infinite  size  vectors  <|>k  according  to  (104)  by  low  pass  filtering  and  scaling 
of  i|/k.  Then  (108)  and  (109)  give  the  extrapolated  estimate  from  the  observations 
(y(m) , - M < m < M). 

Properties  of  Discrete  PSWFs:  We  now  summarize  properties  of  these  functions. 

1.  Let  L be  a low  pass  filter  operator  and  C be  a (2H+1 )x(2M+l ) principal 
minor  of  It,  1 .e. 


A 

Then  the  orthonormal  1 zed  eigenvectors  of  L,  denoted  by  i|*k  form  a complete 
orthonormal  set  of  basis  functions  In  (2M+1)  dimensional  vector  space.  Let 

C*k  - Xk*k  - M < k < M (110) 

Ak>0 

2.  The  discrete  PSWF  are  formed  from  (<|»k)  as  ®xl  vectors  {<j>k>  and  are  defined  as 

$k  ■ -J->  LS^k  - M < k < M (111) 

3.  The  PSWFs  are  orthonormal  over  the  Infinite  Interval,  l.e. 
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i +fc("i)*Jl(">)  ■ «k,t 


(112) 


4.  The  PSWFs  are  orthogonal  over  the  finite  Interval  l.e. 


I = xk6k 

m=-n 


(113) 


5.  The  Fourier  transform  *k(f)  of  (<J>k(m)}  satisfies  the  eigenvalue  Integral 


equation 


s1n(2M+l  )ir(f-f': 
sinir  f-f'T 


\(f')df  “ Wf)‘  |f|<° 


(114) 


The  proof  is  obtained  by  Fourier  transforming  (111)  and  using  (103). 

A 

6.  The  eigenvalues  of  C lie  In  the  Interval  (0,1)  l.e.,  0 < Ak(L)  < 1. 

7.  Let  y(m)  be  a band  limited  signal  whose  spectrum  lies  In  the  Interval 

{-a, a).  If  y (k)  Is  known  over  the  Interval  [-M.M],  then  Its  minimum  norm  least 
squares  extrapolation  estimate  Is  given  by 


. M a. 

y (m)  = l r^-  Mm) , Vm 
k=-M  xk  * 


ak  = l y(m)*k(m) 
m*-M 


(115) 


Note  that  this  gives  y+(m)  » y(m)  for  m e [-M.M]. 

So  far  we  have  considered  only  the  case  when  there  Is  no  noise  In  the 
observed  signal.  In  the  next  few  sections,  we  consider  other  algorithms  where 
additive  noise  or  Interfering  signals  are  allowed  In  the  observations.  The 
algorithms  reduce  to  minimum  norm  least  squares  solutions  In  the  absence  of  any 
noise. 


-43- 


V.  A CONJUGATE  GRADIENT  ALGORITHM  FOR  SIGNAL  EXTRAPOLATION 

In  this  section  we  consider  a slightly  different  but  more  general 
problem  than  that  considered  in  the  previous  section.  We  are  Interested  In 
extrapolation  and  discrimination  of  two  Interfering  signals  (see  Example  1). 

Let  us  assume  that  we  are  given  m observations  which  may  consist  of 
signal,  clutter  and  noise,  where  the  signal  and  clutter  are  bandllmlted  In 
mutually  exclusive  bands  (see  Fig.  1(a)).  The  problem  then  Is  to  obtain 
estimates  of  extrapolated  signal  and  clutter  outside  their  observation 
Interval. 

Let  the  Bandpass  operator  B operate  on  the  signal  In  the  signal  band 
±[f2,f3]  and  the  low  pass  operator  L operate  on  the  clutter  In  the  clutter 


band  t[0,f^]. 

We  Introduce  operator  notation: 
s:  original  signal  (Infinite  vector) 

c:  clutter  (infinite  vector) 

n:  noise  (mxl  vector) 

y:  observed  samples  (mxl  vector) 

B:  bandpass  operator  (Infinite  matrix) 

L:  Low  pass  operator  (Infinite  matrix) 

S:  Selection  operator  (mx«°) 

The  matrices  B and  L are  Toeplitz  matrices  determined  from  the 
sequences  {b^},  respectively,  as  the  Fourier  Inverses  [see  Fig.  11], 

^3  s1n(2irf,k)  - s1n(2irf,k) 

= 2 J cos(2irfk)df  - - ^ ~ (116) 

f2 


,f1 

■ 2j  cos(2trfk)df 


s1n(2irf^k) 


(117) 


L ■ wi-j> 

The  selection  matrix  S,  (Introduced  earlier  also)  Is  defined  as 


5 - to  : im  ; o] 


018) 


where  I Is  an  m x m Identity  matrix.  The  matrix  selects  the  observed  m 
m 

samples  from  an  Infinite  size  vector.  The  observation  vector  can  be 


written  as 


y = Ss  + Sc  + n 


(119) 


Since  s and  c are  bandllmlted  signals  they  satisfy  Bs  « s and  Lc  * c. 


Thus,  we  can  write 


y s SBs  + Sic  + n 


y * S[B  L]  fsl  + n 


(120) 


£ Hx  + n 


(121) 


when  H ■ S[B  L]  and  x 


Now  the  problem  Is  to  find  an  estimate  of  x,  given  y.  In  the 
absence  of  clutter  (c*0),  the  problem  reduces  to  the  extrapolation  problem 
considered  earlier.  The  solution  obtained  by  minimizing  the  least 


squares  norm 


J - ||  y-Hx|| 


Is  given  by 
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1 


x - (HTH)-1HTy 

where  (HTH)-1HT  Is  the  pseudo  Inverse  operator  of  H provided  (HTH)_1  exists. 
However,  this  Is  not  the  case  as  explained  below. 

The  Infinite  matrices  B and  L are  Toeplitz  and  are  diagonalized  by  the 
Fourier  transform  operator.  Therefore, 

F[B  L]FT  = A.  (f ) 

where  F Is  the  Fourier  transform  operator  and  FT  is  its  conjugate  transpose 
andA(f)  Is  as  shown  In  figure  11. 

This  also  implies  that  the  operators  B and  L are  idempotent  i.e., 

B^  = B and  = L.  From  Fig.  11  the  operator  [B  L]  is  singular  which  in 

turn  implies 

STS[B  L]  (123) 

Is  singular,  and  has  a rank  of  atmost  m. 

A nunber  of  approaches  are  once  again  possible  to  find  the  pseudo 
Inverse  of  H,  as  discussed  in  the  previous  section.  Here  we  consider  a two 
step  gradient  method. 

An  example  of  a two  step  gradient  method  where  Initial  convergence  Is 
extremely  rapid  Is  the  Conjugate  Gradient  Method  [12,13]  which  Is  based 
on  the  following  Ideas.  Let  Q denote  the  matrix  HTH  (which  has  dimension 
2Nx2N  to  extrapolate  the  signal  and  clutter  each  to  N points). 

Let  l • 2N.  A set  of  l vectors  (d^  are  conjugate  If 

d/Qdj  - o , i t j . 


Since  Q Is  symmetric  a set  of  such  vectors  exist  and  forms  a basis.  The 
solution  vector  can,  therefore,  be  written  as 

1=1  1 1 

The  .scalers  {exp  and  vectors  (d^  must  be  found  In  a computationally 
feasible  way.  One  way  Is  via  the  algorithm  [13] 


~k+l  ~k  , . 

x = x + akdk 

. -ckTdk 


"kH 


(124) 


vhere  the  {dk>  are  generated  by 


dk+l  = ’Ck+1  + 0kdk 


Ck+lTqdk 


(125) 


dk«dk 


The  vectors  {Ck>  are  the  gradients  of  J at  each  Iteration 
Ck  = Qxk  ‘ HTy  “ Ck-1  + ak-lQdk-l  • 


(126) 


and  the  Initial  conditions  are 


x = y and  d^  * C^. 


(127) 


The  minimum  Is  achieved  In  at  most  & steps,  and  the  method  Is  step  for 
step  better  than  a gradient  method.  A very  attractive  feature  of  this 
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| 

algorithm  Is  that  large  reductions  In  error  are  achieved  In  the 
first  few  steps  [13]. 

I . — 

Looking  at  the  computations  Involved,  we  see  that  except  for  the  single 
matrix  vector  product  Qdfc,  all  vector  operations  Involve  only  i multiplica- 
tions. Hence,  the  order  of  computations  for  each  Iteration  Is  4mN. 

(Note  that  Q Is  composed  from  Toeplltz  matrices,  so  that  advantage  of  FFT 
method  could  be  taken  to  evaluate  Qd^. ) When  Q Is  highly  ill-conditioned, 
or  singular,  the  Iterations  must  be  stopped  at  an  optimum  point.  Alter- 
natively, we  may  add  a small  value  e of  the  order  of  10’®  to  each  diagonal 
term  of  Q.  This  minimizes  convergence  ambiguity  due  to  Ill-conditioning 
and  stabilizes  the  Iterations  greatly. 
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VI.  A MEAN  SQUARE  EXTRAPOLATING  FILTER 

With  the  formulation  and  notation  of  the  problem  of  Section  V,  we 
have* 

y * Hx  + n . (128) 

Now  we  assume  that  x Is  a random  Gaussian  vector  whose  autocorrelation 
matrix  Is  denoted  by  Rx-  The  minimum  mean  square  estimate  of  x Is  given 
by  the  Wiener  filter  as* 

x = [E(x  yT) ][E(y  yT)] _1y  - Gy  . (129) 

Assuming  noise  to  be  independent  of  x,  it  is  easy  to  obtain 


G = RVHT(HR  HT+Rn)_1 

X X \ i 

Rn  = E(n  nT) 


(130) 


which  Is  equivalent  to  the  equations 

<hr/*r„u  ■ y 

s ■ rxhTz 


kx  « E[x  xT]  = E[(*)(sTcT)] 


E(ssT)  E(scT) 
E(csT)  E(ccT) 


(1 40) 


+ We  note  here  the  similarity  between  the  extrapolation  problem  and  the 
restoration  problem  In  Image  processing  [17].  For  example,  an  Image 
* blurred  by  a low  pass  type  operator  H and  contaminated  by  additive  noise 

would  give  rise  to  an  equation  similar  to  (128). 

* Here  E Is  the  expectation  operator. 
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If  signal  and  clutter  are  uncorrelated  (non  overlapping  power  spectra) 


E(ssT)  0 
0 E(ccT) 


(141) 


In  the  special  case  when  the  noise  and  clutter  are  absent,  we  have 


H = B 

R = 0 . 
n 

Thus,  we  have  z ■ (SBRxBTST)_1y.  Since  s(k)  is  a bandllmited 
random  process,  we  must  have 

brxbT  - Rx 

which  gives 

£ = RxST(SRxST)"1y  . 

In  the  worst  case,  when  we  do  not  know  Rx,  we  can  simply  assume  the 
power  spectrum  of  x(k)  to  be  flat  in  its  bandwidth  so  that 


and  R * BST(SBST)+y  where  (SBST)+  is  the  nseudo  Inverse  of  (SBST).  This  is 
the  same  result  as  obtained  by  Cadzow  [See  Eqn.  (95)].  Thus  Cadzow's  one 
shot  method  is  a special  case  of  this  extrapolation  filter.  In  the 
presence  of  noise,  the  extrapolation  filter  estimate  is 

x » BST(SBST+Rn)_1y 

where  (SBS^+Rn)"^  exists  and  is  unique. 
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VII.  A RECURSIVE  EXTRAPOLATION  ALGORITHM 

In  this  section  we  present  a recursive  least  squares  algorithm  based 
on  Kalman  filtering  techniques  where  the  extrapolated  signal  estimate 
Is  updated  recursively  as  a new  observation  sample  arrives. 

Based  on  the  formulation  of  the  problem  as  in  section  V we  rewrite 
equation  (121): 


y = Hx  + n 

The  kth  observation  yk  can  be  written  as 

yk  = hkx  + nk  and  k=0,2,...,m-l  (142) 

where  hT  is  the  kth  row  of  H and  n.  Is  zero  mean  white  Gaussian  noise, 
k k 

The  state  equation  for  the  unknown  extrapolated  vector  x can  be  written 
as 


xk+l  = xk  (143) 

with  Initial  condition  xQ  = x,  where  x is  a random  vector  whose  covariance 
is  given  by 

PQ  * cov(xQ)  - cov(x)  * H ^ (a^}  (144) 

Since  H is  idempotent  i.e.  HH  * H,  it  is  easy  to  verify  that 
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hkht  ■ V* 


Hhk  - hk 


(145a) 

(145b) 


and 


*1"  ■ "kT 


(145c) 


The  Kalman  filter  associated  with  equations  (142)-(144)  is  the  recursive 
least  squares  filter  (e.g.  see  Nahi  [16]) 


*k+l  = \ + Myk"hkV*  *o  = 0 


046) 


th 


where  #k  is  the  k estimate  of  x and  gk  is  the  Kalman  filter  gain. 
The  associated  Riccati  equation  is 


Vi  ■ (I  - rk  pkhkhI»pk(I  - rk  hkhJpk>  * V>>*I  <147> 


and 


ck  = hkPkhk  + 0n2(k)*  an2(k)  = E[nk2] 


9k  = ^ Pkhk 


(148) 

(149) 


Equation  (147)  then  reduces  to 


I 


P,^,  = P.  - — P.  h.hJp. 
k+1  k k k k k 


(150) 


Letting  P^  = H + 9Pk,  and  using  equations  (145), 


9Pk*i  ■ 3Pk  - ^ (I+3Pk>hkhI<I+3P k> 


(151) 


Defining 


h,l  = 9Pkht 


(152a) 


6k,fc  “ ^k,kht 


(152b) 


it  follows  from  (151)  that 


^k+l  £ “ ” c &^^k  * ^k  k ^ * • • • *^sk  *k+l > • . . ,m-1 » (153) 


From  (149)  it  follows  that 


w = 0*^ 


gk  = 5^hk+5k) 


(154) 


From  (145),  (148)  and  (152),  we  get 


ck  Ba0+Cn  (k)  + Bk,k 


(155) 
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Assumlng  that  we  are  basing  our  extrapolation  on  m observed  sample  values  and 

that  we  are  extrapolating  to  N points,  the  maximum  storage  required  is  mN. 

The  major  computation  Is  in  the  calculation  of  the  ^ ^ (153)  and  It  is 

2 

easily  seen  that  the  order  of  computations  involved  is  ~0(m  N). 


VIII.  EXAMPLES,  RESULTS  AND  COMPARISONS 


We  will  now  consider  several  examples  to  study  the  performance  of 
the  foregoing  algorithms.  In  Example  1 below,  we  consider  the  observations 
to  be  of  the  form 

y(k)  = s(k)  + c(k)  + n(k)  , -M  < k < M 

where  c(k)  is  the  bandlimited  clutter  sequence,  and  s(k)  is  a bandlimited 
signal  whose  band  limits  are  known,  i.e., 

S(u>)  = 0 , u) 

2 

n(k)  Is  a zero  mean  white  noise  process  with  variance  on  . 

In  examples  2-9  the  observations  are  of  the  form 

y(k)  s s(k)  + n(k)  ; -M  < k _<  M . 

The  following  examples  have  been  considered. 

la.  y(k)  * s(k)  ♦ c(k)  + n(k)  ; -8  < k < 8 

s(k)  » 1.69s1n(.39irk) 

= 0.382ti 
u>2  * 0.  397it 

clutterband:  a?  * [0,0.  16tt] 
on2  • 0.13 

SCR  - -4. ldB,  SNR  - 19dB 

lb.  y(k)  « s(k)  + c(k);  - 8 < k < 8 

u>i  * . 3tt 

“2 


* .4it 
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us  = .875^2 
s(k)  = 1.69sin(u>sk) 


2.  s(k)  ■ sin(.99o^k)  + s1n( -SSajgk) 


= 0. 8ol>2 
u2  * 2tt/50 

3.  s(k)  = s1n(.99u)2k)  + sin(.85oo2k) 


u>1  = 0 


w2  ■ 0.  Itt 

The  above  two  cases  have  the  same  signal,  but  the  knowledge  of  band- 
limits  Is  different.  In  the  following  cases  we  have  other  signals  with 
different  bandlimits.  In  examples  8 and  9 we  have  additive  noise  also. 

4.  s(k)  = sin(.365irk)  + sin(.385Trk) 


u>1  = - 3tt 
uj2  = . 4tt 


5.  s(k)  * s1n(.365irk)  + sin(.385irk) 


-56- 


6.  s(k) 
“1 


slnf.OBSu^1 
0. 96a)2 


| 


7. 


n 

s(k)  = 0.1 


slnoigk 


a»i  = 0 
^2  = . 1 TT 


slnw  k 

8-  s<k>  - o-1— ^r- 


0)^  = 0 
u2  3 O.lir 
E(n(k) ) = 0,  on2  = 


SNR  - 7.4dB 
9)  s(k)  ■ s1n( . 99uj2k) 

Gd-j  = . 8002 

U)2  ■ 2-TT/50 
E(n(k))  3 0 

aj  * 0.1  , SNR  * 
n 


+ s1n(.975w2k) 


0.01 


+ sin(. 85w2k) 


21 . 6dB 


In  example  la,  application  of  an  eighth  order  autoregressive  model  to 
the  given  data  yields  the  spectrum  shown  In  Fig  1(e).  The  estimate  seems 
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to  be  completely  dominated  by  the  clutter.  A 256-point  DFT-spectrum 
estimate  of  the  data  Is  shown  in  Fig.  1(f),  and  clearly,  this  also  Is 
most  unsatisfactory.  Figures  1(g)  and  1(h)  show  the  signal  extrapolated 
to  199  points  by  the  Conjugate  Gradient  Algorithm  (using  only  10  Itera- 
tions) and  the  Mean  Square  Extrapolation  Filter.  Figures  1(1)  and  l(j) 
show  the  corresponding  results  for  the  extrapolated  clutter.  The  Maximum 
Entropy  method  Is  applied  to  the  extrapolated  signal  and  extrapolated 
clutter  separately  using  a fifteenth  order  model.  Fig.  l(k)  and  l(t)  show  the 
extrapolated  signal  spectra  using  the  Conjugate  Gradient  Algorithm  and  the 
M.S.  Extrapolation  filter.  They  yield  the  signal  peak  at  the  correct 
location.  Figures  l(m)  and  1 (n)  show  the  corresponding  clutter  spectra. 

In  example  lb,  the  signal  bandwidth  is  increased  from  the  previous  case. 

The  actual  spectra  of  signal  and  clutter  are  shown  in  fig.  2(a).  The  FFT 
spectrum  using  256  points  Is  shown  In  fig.  2(b),  and  the  Maximum  Entropy 
Spectrum  using  an  eight  order  model  in  fig.  2(c).  After  extrapolating  signal 
and  clutter  to  125  points  each,  using  the  conjugate  gradient  algorithm  their 
respective  spectra  are  calculated  by  the  Maximum  Entropy  Method  using  a 15th 
order  model.  This  Is  shown  In  figures  2(d)  and  2(e)  respectively.  Figure  2(d) 
shows  a peak  at  the  signal  frequency  along  with  two  subsidiary  peaks  which  are 
seen  to  occur  exactly  at  the  filter  cut-off  frequencies. 


In  examples  2-9  we  consider  either  signal  only  or  signal  with  additive, 
white.  Gaussian  noise. 


Fig.  Ik:  Max  Entropy  Spectrum  of  (1 g)  (15th  order  model)  Fig.  11:  Max  Entropy  Spectrum  of  (Ih)  (15th  order  model) 


nt.X  ENTROPY  OPECTRUtl  Of  EOT  CLUTTER  HC.X  ENTROPY  OPfCTRUfl  OF  EOT  CLJTTER 


Fig.  lm:  Max  Entropy  Spectrum  of  (11)  (15th  order  Model)  Fig.  In:  Max  Entropy  Spectrum  of  (lj)  (15th  order  model) 


-28  ] 


-60 
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The  observed  data  consists  of  17  samples  and  It  is  extrapolated  to 
199  points  using 

1.  Papoulls'  iterative  algorithm 

2.  the  matrix  Ec 

3.  the  matrix  Ec  after  it  has  been  stabilized 

4.  the  conjugate  gradient  algorithm 

5.  the  M.S.  extrapolation  filter  for  Examples  8 and  9 when  the  signal 
contains  noise. 

Extrapolation  using  Papoulis1  algorithm  was  done  with  30  iterations  and  with 
10  iterations  using  the  conjugate  gradient  algorithm. 

Since  all  the  algorithms  ultimately  yield  a minimum  norm  least-squares 
solution,  we  expect  equivalent  results  from  all  the  algorithms.  Figure  3(c) 
shows  the  extrapolation  using  Papoulis'  algorithm,  and  Is  seen  to  give  a 
reasonable  result.  Figure  3(d)  shows  the  extrapolation  via  the  matrix  Ec. 
After  a stabilizing  diagonal  term  of  the  order  of  10”^  is  added  to  the  matrix 
(SLST),  the  extrapolation  obtained  is  as  shown  in  figure  3(e).  A close 
examination  of  figures  3(a),  3(d)  and  3(e)  shows  a slight  phase  shift 
of  the  signal  3(d).  This  may  be  attributable  to  the  111-conditioning 
of  (SLST).  Figure  3(f)  shows  the  extrapolation  by  the  Conjugate  Gradient 
method  after  10  iterations.  This  algorithm  gives  a slightly  Inferior 
extrapolation  compared  to  the  others  in  this  case,  and  perhaps  needs  a few 
more  iterations.  In  later  examples,  however,  it  Is  seen  that  10  Iterations 
are  sufficient. 

In  the  set  of  figures  4,  we  have  the  same  signal  and  observations,  but 
the  uncertainty  In  bandwidth  is  Increased  over  the  previous  case.  The 
relatively  degraded  extrapolations  achieved  now  show  that  there  Is  a 
trade-off  between  bandwidth  uncertainty  and  extrapolation  length,  which  Is 
to  be  expected.  The  set  of  figures  5,6,7  lead  to  similar  conclusions  for 
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signals  with  differing  frequencies  and  different  bandwldths. 

In  the  set  of  figures  8,  we  consider  a sin  x/x  type  of  signal.  The 
phase  distortions  of  the  extrapolated  estimate  in  figure  8(d)  dramatically 
illustrate  the  Ill-conditioned  nature  of  (SLST).  In  the  above  examples 
there  was  no  noise  in  the  observations.  Now,  we  consider  (examples  9 and 
10)  signals  in  additive  zero  mean,  white  Gaussian  noise  at  the  SNR  7.4dB, 
21.6dB  respectively.  The  results  show  that  the  Mean  Square  Extrapolation 
filter,  which  takes  noise  statistics  into  account  produces  the  best 


extrapolation  (Figs.  9(g)  and  10(g))  among  the  algorithms  considered 
experimentally.  Similar  results  are  to  be  expected  form  the  recursive 
extrapolation  algorithm  of  section  VII.  Further  simulations  are  required 
to  study  the  numerical  behavior  of  this  algorithm. 
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Original  Signal  Fig.  5b:  Given  Observations  (17  Sampl 
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Signal  Extrapolated  After  Adding  a Stabilizing  Fig.  5f:  Signal  Extrapolated  by  Conjugate 
Diagonal  Term  to  Matrix  E Gradient  Algorithm 
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Extrapolated  by  Papoulis'  Iterative  Algorithm  Fig.  3d:  Signal  Extrapolated  Via  Matrix 


Fig.  6e:  Signal  Extrapolated  After  Adding  a Stabilizing  Fig.  6f:  Signal  Extrapolated  by  Conjugate 
Diagonal  Term  to  Matrix  E Gradient  Algorithm 


Extrapolated  by  Papoulls'  Iterative  Algorithm  Fig.  7d:  Signal  Extrapolated  Via  Matrix 
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Signal  Extrapolated  After  Adding  a Stabilizing  Fig.  8f:  Signal  Extrapolated  by  Conjugate  Gradient 

Diagonal  Term  to  Matrix  E Algorithm 
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Extrapolated  by  Papoulls'  Iterative  Algorithm  Fig. 
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Flg.  10a:  Original  Signal  Fig.  10b:  Given  Observations  (17  Samples) 
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Fig.  lOg:  Signal  Extrapolated  by  M.S.  Extrapolation  Filter 


IX.  CONCLUSIONS 


The  problem  considered  here  was  to  effectively  discriminate  the 
signal  from  the  Interfering  clutter,  based  on  a small  number  of  observation 
samples.  Conventional  techniques  like  the  DFT  and  Maximum  Entropy  Methods 
are  seen  to  yield  poor  results. 

Given  that  the  signal  and  clutter  are  band  limited  In  mutually  exclusive 
bands,  a good  method  of  Improving  resolution  is  to  extrapolate  the  signal 
outside  the  observation  interval  and  then  estimate  the  spectrum.  Though 
continuous  time  band  limited  signals  can  be  extrapolated  exactly  outside  the 
observation  Interval,  this  Is  not  possible  In  the  discrete  time  case.  In 
fact.  In  this  report  we  have  proved  that  the  discrete  extension  of  the 
continuous  algorithms  leads  to  an  extrapolated  signal  which  Is  optimum 
In  a minimum  norm  least  squares  sense. 

We  have  introduced  some  new  algorithms  for  signal  extrapolation,  viz.; 

a)  Conjugate  Gradient  Algorithm 

b)  Mean  Square  Extrapolation  Filter 

c)  Recursive  Extrapolation  Filter 

d)  An  Extrapolation  Algorithm  via  Discrete  PSWFs. 

The  Conjugate  Gradient  method  Is  an  Iterative  technique  which  has 
a rapid  Initial  rate  of  convergence  and  hence  has  an  obvious  advantage 
over  Papoulls'  algorithm  which  has  a linear  rate  of  convergence.  The  Mean 
Square  extrapolation  filter  Is  a non-iterative  method  that  takes  noise 
statistics  Into  account  and  simultaneously  filters  the  clutter  from  the 
signal.  Cadzow's  one  shot  method  Is  seen  to  be  a special  case  of  this 
filter. 


Further  experiments  have  to  be  performed  for  the  Recursive  Extrapolation 
filter  (which  also  considers  noise  statistics)  and  for  extrapolation  via 
Discrete  PSWFs.  In  practice,  extrapolation  Is  achieved  only  to  a limited 
. extent  beyond  the  observation  Interval  and  depends  on  the  signal  bandwidth 

uncertainty.  The  larger  this  bandwidth  uncertainty,  the  smaller  the  length 
to  which  the  signal  can  be  extrapolated.  Although,  In  the  absense  of  noise, 
all  algorithms  yield  a minimum  norm  least  squares  solution,  their  Implementations 
are  different  and  lead  to  different  truncation  errors.  Such  error  analysis 
and  other  numerical  features  for  the  extrapolation  problem  need  further 
work.  Our  experiments  show  that  whatever  the  uncertainty  of  the  signal 
bandwidth  might  be,  extrapolation  of  the  signal  followed  by  a spectral  estimation 
Improves  the  spectral  estimate. 
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Table  1 


METHOD 

COMPUTATIONAL 

COMPLEXITY 

GENERAL 

COMMENTS 

MAX.  ENTROPY  or 
Autoregressive  (Burg, 
Parzen  & others) 

pxp  Toeplitz  Eqns. 

+ FFT  operation 
= 3p2  + O(NlogN) ; 
p«N 

1.  Simple  linear  Eqns.,  easy  to 

Implement. 

2.  Good  results  for  all -pole  spectra. 

3.  Falls  In  the  presence  of  noise  and 
clutter-unless  a large  number  of 
observation  samples  are  available. 

4.  Order  of  the  model  difficult  to 
select. 

5.  Performance  Improved  by  applying  It 
on  extrapolated  signal. 

Continuous  PSWF 
(Sleplan  et  al . ) 

Very  large.  Func- 
tions extremely 
difficult  to  cal- 
culate 

1.  For  extrapolation  of  bandllmlted, 
continuous  signals;  existence  guaran- 
teed. 

2.  Extremely  difficult  to  Implement. 

3.  Noise  sensitive 

4.  Useful  In  establishing  existence, 
uniqueness  & other  properties. 

Iterative 

Extrapolation 

(Papoulls) 

0(4Nlog2N)  real 
operations  per 
Iteration 

1.  Easy  to  Implement 

2.  Is  a gradient  algorithm  with 

linear  convergence.  Requires  a large 
number  of  Iterations  and  FFT 
operations  at  each  Iteration. 

3.  Does  not  take  Into  account  noise 
statistics. 

Extrapolation 

Matrix,  Eoo 
(Sabri  and 

Steenaart) 

If  observed  data  * 

2 M+l  and  extrapo- 
lated length  * N 
then  ~ 0(N3)  to  In- 
vert NxN  Matrix 
+1/2(2M+1 )(N-2M-1 ) 
operations  to  find 
extrapolated  signal 

1.  Eoo  does  not  exist.  EN  exists. 

2.  A large  (NxN)  Ill-conditioned 
matrix  has  to  be  Inverted. 

3.  Noise  sensitive.  Can  be  stabilized  by 
adding  a diagonal  term  to  G. 

4.  Noise  statistics  not  considered. 

E (Cadzow) 

If  observed  data  * 
m pts  extrapolated 
to  N pts, 
0(3nr+irr+mN) 

1.  Easy  to  Implement,  If  Er  Is  stable. 

2.  An  Ill-conditioned  matrix  (mxm 
Toeplitz)  has  to  be  Inverted. 

3.  Noise  sensitive.  Can  be  stabilized. 

4.  Noise  statistics  not  considered. 

Conjugate 

Gradient  (Jain  & 
Ranganath) 

- 0(2mN)  operations 
per  Iteration 

1.  Easy  to  Implement 

2.  Extremely  rapid  Initial  rate  of 
convergence 

3.  Small  number  of  Iteration  required 

In  practice 

4.  Noise  sensitive,  but  can  be  stabilized 

5.  Noise  statistics  not  considered 

Table  1,  continued 


GENERAL 

COMMENTS 


Easy  to  Implement 

An  mxm  Toeplitz  matrix  has  to  be 

Inverted. 

Takes  Noise  statistics  Into  account 
Reduces  to  Er  In  the  noise  free  case 


Easily  implemented,  updates  extra- 
polated estimate  as  new  data  arrives 
Takes  noise  statistics  Into  account. 


More  difficult  to  Implement  than 
(7)  or  (8).  Easy  to  Implement  once 
the  eigenvectors  have  been  computed 
Noise  statistics  not  considered. 
Accuracy  depends  on  the  accuracy  of 
eigenvalues  and  eigenvectors. 


METHOD 

COMPUTATIONAL 

COMPLEXITY 

Mean  Square 
Extrapolation 

Filter  (Jain  A 
Ranganath) 

~ (BnAm^+mN) 
operatl ons . mN 
operation  once 
Filter  gain  has 
been  computed. 

Recursive  Extra- 
polation (Jain  & 
Ranganath) 

N operations 
per  data  point. 

If  gains  are  pre- 
computed. 

Discrete  PSWF 

Singular  Value 
Expansion 
(Papoulls,  Jain  & 
Ranganath) 

Requires  solving 
for  eigenvalues 
and  eigenvectors 
of  a (2M+l)x(2M*l) 
Toeplitz  matrix,  a 
low  pass  filtering 
operation  and  a 
finite  series 
expansion. 
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